release: Production-readiness refactor, S3-backend, and dependency updates#189
Conversation
Bumps [roc-validator](https://github.com/crs4/rocrate-validator) from 0.8.1 to 0.9.0. - [Release notes](https://github.com/crs4/rocrate-validator/releases) - [Changelog](https://github.com/crs4/rocrate-validator/blob/develop/CHANGELOG.md) - [Commits](crs4/rocrate-validator@0.8.1...0.9.0) --- updated-dependencies: - dependency-name: roc-validator dependency-version: 0.9.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [requests](https://github.com/psf/requests) from 2.33.0 to 2.33.1. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](psf/requests@v2.33.0...v2.33.1) --- updated-dependencies: - dependency-name: requests dependency-version: 2.33.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
…-0.9.0 Bump roc-validator from 0.8.1 to 0.9.0
Bump requests from 2.33.0 to 2.33.1
Makes MinIO an opt-in and disabled by default, via a MINIO_ENABLED setting. When disabled the service stores nothing and only the stateless validation endpoint is exposed (the ID endpoints are not registered) - config: add get_bool_env helper and MINIO_ENABLED (default False) - routes: split the POST blueprint so /validate_metadata is always available; register the ID-based POST/GET routes only when MINIO_ENABLED is true - compose: move the minio service behind a `minio` profile and drop it from depends_on; MINIO_ENABLED into the flask/celery services - docs/env: document MINIO_ENABLED and the --profile minio workflow - tests: cover enabled and disabled modes; integration tests now starts the minio profile with MINIO_ENABLED=true
Co-authored-by: Eli Chadwick <eli.chadwick@manchester.ac.uk>
feat: Make MinIO object storage optional (#156)
- Replace the import-time Config classes with a validated Settings dataclass built in create_app(). - Fails early with an aggregated error when storage is enabled but the required S3/Celery vars are missing. - Gate storage routes on settings.storage_enabled; Settings injected in tests instead of monkeypatching. - Add app/storage as a "vendor-neutral" abstraction (so, works with MinIO, RustFS, or any other S3 service) - Closes #171, #172
Implement S3Backend using boto3. Supports MinIO SDK features such as pagination. Includes tests Refs #172
Add app/crates/ids.py: validate_crate_id / is_valid_crate_id and an InvalidCrateId exception. IDs are constrained to a safe charset (start alphanumeric, [A-Za-z0-9._-], max 128, no '/' or '..'). Fixes the ".zip in crate ID" fragility. IDs are no longer parsed. Closes #174
Add app/crates/layout.py (separate crate/result key prefixes) and resolver.py. `resolve_crate()` locates a crate by direct stat checks on "canonical keys" instead of prefix-listing. - Distinguishes zip vs directory, - confirms ro-crate-metadata.json for directories, and - reports "AmbiguousCrate" / "CrateNotFound" - Fixes sibling false-match, .zip-in-id, and some missing-metadata gaps. Closes #175
- Add `app/validation/results.py` (`ValidationOutcome` with valid/invalid/error status, detail/error, serialisation) and `runner.py` wrapping rocrate_validator so both entry points always return an outcome; - Replaces the ValidationResult|str / isinstance pattern. Closes #176
- replaces metadata path with `run_metadata_validation()`, which validates inline via the runner and returns a `ValidationOutcome` (200 valid/invalid, 422 unvalidatable). - remove the metadata Celery task and `perform_metadata_validation`, which addresses the result.get() blocking and the return-in-finally bug. closes #169
- Move POST/task/GET onto StorageBackend + resolver + runner with server-side credentials. POST resolves before queueing (400/404/409/503 via app error handlers); - the Celery task (`run_validation_job`) runs fetch->validate->persist->webhook with retries and persists error outcomes; GET reads the results prefix. - Add s3_crate_prefix/s3_results_prefix to Settings. Delete minio_utils and test_minio, remove minio from requirements and recompile the lock. - Covers the store-backed validation flow Closes #177, #178, #179
- Refactor `send_webhook_notification` to retry failures with exponential backoff, add a request timeout, and raise `WebhookDeliveryError` on terminal failure instead of accepting it. - Results are persisted so a delivery failure can be surfaced without losing the result. Closes #180
Rewrite logging_service with a JsonFormatter, a RedactionFilter that masks configured S3 credentials, and a RequestIdFilter closes #177
- Adds endpoints to check if the S3 is reachable and Celery / broker connected; - Returns 503 with a per-check breakdown when a dependency is down. - When storage is disabled both checks report "disabled". Closes #181
- Consolidate packaging and tooling into pyproject.toml and standardise linting and formatting with ruff. - Move metadata and deps into pyproject.toml; compile requirements.txt (runtime) and requirements-dev.txt (runtime + dev: pytest, pytest-mock, moto, ruff) from it via pip-tools; retire requirements.in. - Move pytest config into [tool.pytest.ini_options] and remove pytest.ini. - Add [tool.ruff] config; apply ruff check --fix and ruff format across code. - CI: unit_tests installs requirements-dev.txt; add a lint workflow running ruff check and ruff format --check on PRs to develop. Closes #183, #173
- Replace the MinIO dev container with a vendor-neutral "objectstore" service running rustfs/rustfs. - Update .env/example.env to S3_* app config plus RUSTFS_* container credentials; the app/worker stay vendor-neutral (S3_* only). - Start with `docker compose --profile objectstore up`.
- Move unit tests into app-mirroring sub-packages (storage/, crates/, etc.) and switch pytest to importlib import mode with pythonpath=".". - Rewrite the integration test for the new stack - Update the integration workflow deps Closes #184
… and image A few final things to bundle together: - bump `roc-validator` 0.9.0 -> 0.10.0 to use offline/cache/extra-profiles - rename the WSGI entrypoint and align image/label naming. - Add `VALIDATION_OFFLINE`, `CACHE_PATH` and `EXTRA_PROFILES_PATH` settings - Offline validation is opt-in (default off). - `extra_profiles_path` ADDS to the bundled profiles (roc-validator 0.10.0 fix) - Warm validator HTTP cache at image build (rocrate-validator cache warm) - Replace the site-packages profile bake with extraction to /app/extra-profiles loaded via `EXTRA_PROFILES_PATH` - This is passed as a build arg from the profiles workflow. - Rename entrypoint `cratey.py` -> `wsgi.py` to fit convention - Rename the published image and the five-safes label to ro-crate-validation-service; point image.source at the renamed repo. Closes #162, #163, #164
- Addresses review feedback on #186 - Gives both POST blueprints a shared tag so the Swagger docs show a single POST group whether or not the store-backed routes are registered - APIFlask emits one top-level tag entry per blueprint, so the shared tag would appear twice in openapi.json (OpenAPI requires unique tag names) - Added a spec processor decorator to depulicate list. - Added supporting test
- Addresses review feedback on #186 - Redo profile-fetch and cache-warm RUN commands as BuildKit heredocs - Added set -e to keep old && failure semantics - proposed heredoc blocks would only report exit status of its last command
- Addresses review feedback on #186
- Addresses review feedback on #186 - Idea is that it pins every action to the commit SHA of the latest release in its current major
Production-readiness refactor: drop MinIO, make Crate IDs more robust, add async validation and offline cache
Bumps [redis](https://github.com/redis/redis-py) from 7.4.0 to 8.0.1. - [Release notes](https://github.com/redis/redis-py/releases) - [Changelog](https://github.com/redis/redis-py/blob/master/CHANGES) - [Commits](redis/redis-py@v7.4.0...v8.0.1) --- updated-dependencies: - dependency-name: redis dependency-version: 8.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>
chore(deps): bump redis from 7.4.0 to 8.0.1
Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.6.3 to 2.7.0. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst) - [Commits](urllib3/urllib3@2.6.3...2.7.0) --- updated-dependencies: - dependency-name: urllib3 dependency-version: 2.7.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>
chore(deps): bump urllib3 from 2.6.3 to 2.7.0
Bumps [roc-validator](https://github.com/crs4/rocrate-validator) from 0.10.0 to 0.11.2. - [Release notes](https://github.com/crs4/rocrate-validator/releases) - [Changelog](https://github.com/crs4/rocrate-validator/blob/develop/CHANGELOG.md) - [Commits](crs4/rocrate-validator@0.10.0...0.11.2) --- updated-dependencies: - dependency-name: roc-validator dependency-version: 0.10.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
…-0.10.0 chore(deps): bump roc-validator from 0.10.0 to 0.11.2
Bumps [idna](https://github.com/kjd/idna) from 3.10 to 3.15. - [Release notes](https://github.com/kjd/idna/releases) - [Changelog](https://github.com/kjd/idna/blob/master/HISTORY.md) - [Commits](kjd/idna@v3.10...v3.15) --- updated-dependencies: - dependency-name: idna dependency-version: '3.15' dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>
chore(deps): bump idna from 3.10 to 3.15
- requests 2.34 made `RequestsCookieJar` a `TYPE_CHECKING`-only import in requests/models.py, which breaks requests-cache 1.2.1 - Dependabot's #157 bumped only requests and failed the docker build - I also updated to drop async-timeout (redis only needs it on Python < 3.11.3) and typos (no longer required by roc-validator).
chore(deps): bump requests to 2.34.2 and requests-cache to 1.3.3
|
Is it worth putting together an upgrade guide? answering questions like:
thinking of how we can make this update easy for folks like @AnnieZQH and K8TRE team to adopt. It would be particularly useful (and maybe sufficient) if all new/changed settings could be listed in our release documentation |
Due to release timings and deadlines, this will be an addition to the docs once the site is up and running and pointed to by https://github.com/Health-Informatics-UoN/analysis-types-for-TREs/blob/master/observablehq.config.js#L47 |
Summary
Merges
developintomainfor the next release. The bulk of this is the production-readiness refactor (#186), plus the optional-MinIO work (#161) and a series of dependency updates. All of which have been reviewed.Features
/healthzand/readyzValidationOutcometype and validation runner; metadata validation is now synchronousS3Backend; object storage is optional (Make MinIO storage optional #156)Breaking changes
ghcr.io/esciencelab/ro-crate-validation-service)minio-pydropped for generic boto3S3BackendTooling & CI
pyproject.tomland ruff; consolidated Dockerfiles and compose filesapp/layoutDependency updates